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ABSTRACT 



This paper examines how the process of soliciting evaluation 
feedback from nonresident students in the Army Management Staff College 
(Virginia) program on leadership and management for civilian employees of the 
Army has evolved since 1995. Course design is briefly described, including 
the use of video-teleconferences , chat rooms, an electronic bulletin board 
and library, and e-mail. Discussion of the evaluation plan covers 
administration and response nonresident class surveys from 1995 through 1998, 
graduate surveys, and supervisor surveys. The analysis and electronic 
dissemination of survey data to staff and faculty is also summarized. (AEF) 
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Introduction 



The Army Management Staff College offers a resident course in leadership and management 
for civilian employees of the Army. The course is designed to provide senior civilian leaders 
with the skills, knowledge and understanding of the "Big Picture" required to make critical 
decisions in the workplace of today and prepare them for senior level positions within the 
Department of the Army or other government agencies. Since 1995 the college has offered a 
vr one year nonresident version of the program. Since the inception, determining how best to 

00 solicit evaluation feedback from the students has been a challenge. This paper will examine 

cn how the process has evolved since 1995. 

1 

Course Design 

Students are divided into 5 seminars with approximately 16 students and a full-time faculty 
leader. Seminar members live all over the United States, as well as Korea, Japan, and 
Europe. Students attend a one-week resident session at the beginning of the program and 
another one-week session at the conclusion. In between, instruction is primarily paper 
based. The content is divided into 4 segments with graded assignments required for each 
segment. There are two video-teleconferences during the year with videos provided to those 
students unable to attend the sessions. Students have also been provided access to chat 
rooms, an electronic bulletin board and library, as well as e-mail to supplement the paper 
materials. Most seminars schedule weekly study sessions in the chat room with logs being 
posted in the library for those unable to attend. 



Evaluation Plan 

From the beginning the plan was to evaluate the nonresident program the same as the 
resident program. Student surveys be administered for at the end of each resident session 
and periodically throughout the year. In addition, a graduate survey would also be 
distributed 6 months after graduation and a survey sent to supervisors one year after 
graduation. One of the advantages of using the same evaluation strategy is the ability to 
compare data on the graduates of both programs to determine if supervisors and graduates 
themselves respond differently. 
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Student Surveys 

Nonresident class 95. For NR 95, surveys for the 2 resident sessions were administered using 
paper and Scantron forms. Surveys were administered at the conclusion of each of the 4 
segments of the curriculum. For the first survey, many of the questions were taken from 
surveys developed for the resident program. However, analysis of the responses showed a 
definite difference in responses between resident and nonresident students on the same 
items. In addition, responses that had not been anticipated were needed for some questions. 
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An example is the questions on the quality of feedback provided to students on their 
assignments. While the concept was for faculty in a week or two, this was not happening 
many students had not received feedback on all assignments prior to the survey. As a 
result the questions were modified for the second survey and focused on whether the 
program was working as planned (i.e., students receiving feedback from one assignment 
prior to the submission of the next) as well as the quality of the methods, materials, and 
experience. 

Prior to the second survey, students obtained access to the electronic facilities. Therefore, the 
survey was also available for them to download form the library as well as being mailed td 
them. They could respond electronically or via mail. The same procedures were used for the 
third and fourth survey. Only 2 or 3 students opted to submit their results electronically. The 
fourth survey was due about 2 weeks before the final resident session. The response rate for 
this survey was only 35%, far lower than any of the other three where response rates ranged 
from 72% to 58%. 



Nonresident class 96. Surveys were again administered for each of the resident sessions 
using paper and Scantron forms. The number of surveys administered while the students 
were off-site was reduced from four to three. Because of the low response rate for the final 
survey of NR 95, the decision was made to include questions on the final segment of the 
curriculum completed off-site in the survey for the second resident session. In addition, the 
survey administration changed from paper-based to disk. Each student was mailed a disk 
with the survey on it. They completed the survey and mailed it back in a post-paid disk 
mailer. While the return rates using this method were comparable to the paper-based 
surveys of NR 95, there were problems. The only software available that could handle open- 
ended responses was DOS based while most students were more comfortable with a 
Windows environment. In addition, one student had a Macintosh that the software would 
not run on. This problem was resolved by mailing a paper copy to the one student. During 
each administration, a number of diskettes were damaged during mailing. While 99% of the 
data was recovered, the process was time-consuming. 



Nonresident class 97. Again, -the resident session surveys were administered using paper 
and Scantron forms. However, the three off-site surveys were administered via the World 
Wide Web. The surveys were attached to the AMSC web site but could only be accessed 
with the URL. Once the survey was in place, each student was sent an e-mail with the URL 
address and the date when the survey would no longer be available. The responses were fed 
into a database that could then be entered into statistics software for analysis. However, 
return rates using the web are lower than for the other methods with the exception of the 
fourth survey. For the fourth and fifth surveys a competition was implemented between 
seminars with the results posted periodically during prior to the due date (Table 1). 

Student response to the competition was very positive and the concept will be continued for 
the next nonresident program. During the administration period for the second survey, 
there were problems with Internet access to Fort Belvoir. As a result, the AMSC server was 
down for several days. Once the connection was restored, problems continued with sporadic 
interruptions of service for several days. In an effort to determine why responses rates were 
so low for the early surveys, questions on the web survey were added to the final survey. 
Unfortunately the responses provided little explanation. Most, 71%, of the students said they 
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had no problems accessing the survey at the web site and there were few recommendations 
for improving the process. 



Table 1. Return Rates for Student Survey 





NR 95 




NR 96 




NR 97 




Method 


Return 


Method Return 


Method Return 


Survey 1 

(Resident Session) 


Paper 


92% 


Paper 


92% 


Paper 


92% 


Survey 2 


Paper 


72% 


Disk 


71% 


Web 


51% 


Survey 3 


Paper 


64% 


Disk 


65% 


Web 


52% 


Survey 4 


Paper 


58% 


Disk 


59% 


Web 


70% 


Survey 5 


Paper 


35% 


Combined with 6 


Combined with 6 


Survey 6 

(Resident Session) 


Paper 


91% 


Paper 


96% 


Paper 


99% 



Nonresident class 98. Resident sessions will again be paper based and the web will continue 
to be the primary method of administration. However, students who can not easily access 
the web or are not comfortable using this method will have the option of completing the 
survey via e-mail. Software has been purchased that allows for the development of a survey 
to be delivered via web or e-mail and consolidation of results from both methods into a 
single database. In addition to adding e-mail capabilities, it eliminates the requirement for a 
programmer to be involved in the process. 

Graduate Surveys 

Graduates of the first two nonresident programs were surveyed 6 months after graduation 
the same as graduates of the resident course. The timeframe for administering graduate 
surveys is being changed for classes graduating in 1998. Surveys will be administered for 
both resident and nonresident program graduates in September 1999. Because of the nature 
of the program, graduates continue to receive surveys periodically after the initial graduate 
survey at 3 to 5 year intervals. 

The questions dealing with the usefulness of the topics taught in the course and perceptions 
of how well AMSC has prepared them for the workplace were the same as those 
administered to resident course graduates. Questions dealing with the design and focus of 
the course were modified to reflect the differences in the delivery of the courses. However, 
the final result is six questions were removed for the nonresident graduates and three new 
ones added. 

In general, while the responses from the nonresident graduates in NR 95 and NR 96 are 
positive, they are lower than those from resident course students. This is especially true for 
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items dealing with how well the course prepared them to think critically and make 
decisions, two areas that are difficult to replicate in the nonresident environment. 



Supervisor Surveys 

Like the resident course, the supervisors of nonresident graduates are surveyed one year 
after graduation. Supervisors are asked to evaluate the knowledge of the graduate on 
subjects taught in the curriculum as well as the potential of the graduate for senior level 
positions. The questions are exactly the same as those used for the resident program. 

The results for almost all items are higher from supervisors of nonresident graduates. Only 
one item, which deals with the graduate's ability to manage change, was rated lower for the 
nonresident class (Table 2). 

Table 2. Comparison of Supervisor Responses 



Survey Item: The graduate understands how the organization manages change. 





95-3 


96-1 


96-2 


NR95 


Strongly Agree /Agree 


83% 


82% 


96% 


75% 



Results 

A report is prepared for each survey administered. Copies are distributed electronically to 
the Dean of Academics, Commandant, and Department Chairs. In addition, a copy is placed 
on the LAN where it is accessible to all staff and faculty. Course developers and college 
management has been encouraged not to react to the responses from a single survey. Instead 
they should look for trends across classes and this type of information is contained in the 
report when appropriate. In addition, responses from the various sources are compared. The 
graduate surveys are compared to the results of the student surveys for the class. Results 
from Nonresident surveys are compared to the comparable resident group. Results from 
supervisors are compared to those of the graduates on similar items. By looking at 
similarities and differences between multiple sources, a better picture of the effect and 
impact of each course is available. 
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